Sonification of Surface Tapping: Influences on Behavior, Emotion and Surface Perception

Interaction sounds when tapping on a surface provide information about the material of the surface and about one’s own motor behaviour. With the current developments in interactive sonification, it is now possible to digitally change this audio-feedback resulting from object interaction. Here we evaluated a model for a sonic interactive surface. This model uses a system capable of delivering surface tapping sounds in real-time, when triggered by the users’ taps on a real surface or on an imagined, “virtual” surface (i.e., when tapping in the air). Across different conditions the audio-feedback was varied so that the heard tapping sounds corresponded to different applied strength during tapping. We evaluated the effect of the altered tapping sounds on (1) emotional action-related responses: perceived effort and aggressiveness when tapping on the surface, emotional valence, dominance, and arousal measured through self-report and biosensors, (2) participants’ way of interacting with the surface: maximum acceleration and frequency of tapping movement, and (3) surface perception: perceptual quality of hardness. Results show the influence of the sonification of surface tapping at all levels: emotional, behavioral and perceptual. We conclude by addressing some implications of our results in the design of interactive sonification displays and tangible auditory interfaces aiming to change perceived and subsequent motor behaviour, as well as perceived material properties.